Performance evaluation of SARS-CoV-2 rapid diagnostic tests in Nigeria: A cross-sectional study

The COVID-19 pandemic challenged health systems globally. Reverse transcription polymerase chain reaction (RT-PCR) is the gold standard for detecting the presence of SARS-CoV-2 in clinical samples. Rapid diagnostic test (RDT) kits for COVID-19 have been widely used in Nigeria. This has greatly improved test turnover rates and significantly decreased the high technical demands of RT-PCR. However, there is currently no nationally representative evaluation of the performance characteristics and reliability of these kits. This study assessed the sensitivity, specificity, and predictive values of ten RDT kits used for COVID-19 testing in Nigeria. This large multi-centred cross-sectional study was conducted across the 6 geo-political zones of Nigeria over four months. Ten antigen (Ag) and antibody (Ab) RDT kits were evaluated, and the results were compared with RT-PCR. One thousand, three hundred and ten (1,310) consenting adults comprising 767 (58.5%) males and 543 (41.5%) females participated in the study. The highest proportion, 757 (57.7%), were in the 20–39 years’ age group. In terms of diagnostic performance, Lumira Dx (61.4, 95% CI: 52.4–69.9) had the highest sensitivity while MP SARS and Panbio (98.5, 95% CI: 96.6–99.5) had the highest specificity. For predictive values, Panbio (90.7, 95% CI: 79.7–96.9) and Lumira Dx (81.2, 95% CI: 75.9–85.7) recorded the highest PPV and NPV respectively. Ag-RDTs had better performance characteristics compared with Ab-RDTs; however, the sensitivities of all RDTs in this study were generally low. The relatively high specificity of Ag-RDTs makes them useful for the diagnosis of infection in COVID-19 suspected cases where positive RDT may not require confirmation by molecular testing. There is therefore the need to develop RDTs in-country that will take into consideration the unique environmental factors, interactions with other infectious agents, and strains of the virus circulating locally. This may enhance the precision of rapid and accurate diagnosis of COVID-19 in Nigeria.


Introduction
Coronavirus disease 2019 (COVID- 19), which was declared a pandemic by the World Health Organisation (WHO) in March 2020, is a crisis that has challenged health systems globally.Over 774 million infections and 7 million deaths have been documented worldwide as of January 14, 2024, with about 9.60 million cases occurring in Africa [1].Early and rapid diagnosis is crucial for rapidly identifying and isolating infected individuals in a bid to slow down transmission, provide timely clinical management to those affected, and protect health systems operations through triaging at admissions [2].
Reverse transcription polymerase chain reaction (RT-PCR), a very sensitive and specific technique, is the gold standard testing method for detecting the presence of SARS-CoV-2 in clinical samples [3].However, it is resource-intensive requiring huge investments, including highly skilled manpower, expensive machines, elaborate specimen transport systems, and costly reagents and consumables [4].More importantly, RT-PCR techniques cannot be implemented in remote settings in Africa, including Nigeria, due to a lack of appropriate infrastructure [2,5].
With the frequent emergence of variants of the virus, which have fuelled new outbreaks of the pandemic [6], coupled with low vaccine uptake and coverage [7], the world is faced with a looming possibility that COVID-19 will persist, transitioning from a pandemic to an endemic disease [8].Diagnostic platforms that are sustainable for the long haul are necessary, to maintain widespread accessibility and inform public health decision-making.Rapid diagnostic tests (RDTs), such as antibody (Ab) and antigen (Ag) RDTs, are attractive alternatives to the technically demanding, resource intensive, though highly sensitive molecular platforms for several economic, public health, and operational reasons [5].For example, from the economic perspective, RDTs are cheaper to produce and mostly do not require power to store reagents and conduct tests.The World Health Organization recommends deploying rapid tests with a sensitivity of at least 80% and specificity greater than 97% compared to the RT-PCR in applicable settings [9].Currently, the diagnostic performance of most RDTs in the market is yet to be adequately evaluated in field situations in many African settings.Despite the near-universal roll-out of these tests, real-world data remains limited to evaluations conducted in Kenya [10], Uganda [11], South Africa [12], Cameroon [13], and Ghana [14].Since the country's third wave of COVID-19 in July of 2021, WHO and the Nigeria Centre for Disease Control (NCDC) recommended kits have been deployed to scale up testing in Nigeria.However, their performance and those of other commercially available kits are yet to be evaluated in a nationally representative field study to ascertain their reliability.The role of accurate and reliable diagnosis in the management and control of an infection cannot be overemphasised.This prospective study was therefore conducted to assess the sensitivity, specificity, and predictive values of RDTs available for point-of-care COVID-19 diagnosis in Nigeria.

Study area/design
Seven teams, coordinated by the Nigeria COVID-19 Research Coalition (NCRC), designed a harmonised protocol to evaluate the performance characteristics of available antigen and antibody SARS-CoV-2 rapid diagnostic kits (RDTs) in Nigeria.This multi-centre cross-sectional study was conducted in thirteen (13) states and the Federal Capital Territory (FCT), within the six (6) geo-political zones of Nigeria (Table 1).It was carried out for four months (October 2021 to January 2022) and spanned through two successive waves (3 rd and 4 th ) of the COVID-19 pandemic.Government-approved testing centres were used to collect samples from consenting participants suspected of COVID-19.
A total of six antigen RDT kits were evaluated, including-  2).This list was based on the National Agency for Food and Drug Administration and Control (NAFDAC) recommendation.

Study participants
Consecutive consenting adults (18 years and above) who presented at the various centres for COVID-19 testing were enrolled in the study over four months (October 2021 to January 2022).All sites followed the same inclusion and exclusion criteria as shown in the flowchart in Fig 1.

Sample size determination
The sample size for the study was determined using RT-PCR as a reference test and the following assumptions: a pre-determined sensitivity value of 70% and specificity value of 99%; the positivity rate of COVID-19 among suspected cases in Nigeria of 10%; maximum marginal error of 5%; and an alpha level of 0.05.The sample size for the study was determined using the formula [15][16][17]  where P1 and P2 denote the sensitivity of two alternative point-of-care diagnostic tests respectively for testing the hypothesis.P⁻ is the average of P1 and P2, Zα is the standard normal deviation at a 95% confidence interval (1.96), and Zβ is the statistical power to detect a difference of 10%.Based on the report of a previous study [18], P1 was 88.7% (sensitivity) and 90.6% (specificity) using IgM/IgG based diagnostic test.The calculated sample sizes were 130 and 96 test per point-of-care diagnostic test for each of the 7 study sites based on the sensitivity and specificity assumptions respectively.A minimum sample size of 130 tests per point-of-care diagnostic test for each of the 7 study sites, which gave a total minimum sample size of 910 was adopted.

Sample collection and processing
Concomitant nasopharyngeal, oropharyngeal, and nasal swabs as well as whole blood samples, were collected from each consenting individual attending the various sample collection centres for COVID-19 testing.Samples were collected following strict infection prevention and control protocols as outlined in the NCDC specimen collection guidelines.The nasopharyngeal and oropharyngeal swab samples were collected and placed into a viral transport medium (VTM) and stored at 2-7˚C before transportation to the molecular laboratory for RT-PCR testing.The nasal swabs were collected and placed in a diluent supplied by the manufacturer of the respective antigen test kits.In addition, 10 ml of blood was collected from each participant into EDTA or plain collection tubes.Samples were transported in cold-box containing ice packs to the laboratories, where they were centrifuged at 3500 rpm for 15 minutes.The plasma or serum was separated and then tested for the presence of SARS-CoV-2 antibodies using RDT kits according to manufacturers' instructions.Thereafter, aliquots of the samples were shipped to biorepositories for storage and future SARS-CoV-2-related studies.The schema of sample collection and testing is shown in Fig 2.

Reference method
RT-PCR was used as the gold standard.Viral RNAs were extracted from the naso-/oro-pharyngeal swabs using approved kits supplied by NCDC (the kit supplied was based on what was available at the time) and following the manufacturer's instructions.Table 3 shows the list of PCR reagents supplied by NCDC during this period.The presence of SARS-CoV-2 in each

Data collection and analysis
Individuals who consented were enrolled in the study; data from all centres were pooled together and statistical analysis was performed for all participants with complete information.Socio-demographic information obtained from study participants and the laboratory results were entered into the research electronic data capture (REDCap) tool.Point estimates of sensitivity, specificity, positive predictive values, negative predictive values, and the area under the receiver operating characteristic curve was used to determine the general performance and characteristics of the RDTs.Participants with indeterminate results on either RDT or PCR reference tests as well as those with missing data were excluded from the final analysis.Statistical analysis was done using StataCorp.2021.Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC.The area under the receiver operating characteristic curve (AUROC) was computed for individual test kits using the sensitivity and 1-specificity of each test to plot a curve across varying cut-off values [19].AUROC with values > 0.70 were adjudged to possess good to excellent diagnostic capacity, whereas AUROC values < 0.5-0.70 were adjudged to have poor diagnostic accuracy.AUROC values less than 0.5 imply extremely poor discriminative ability [20,21].
The agreed criteria for percentage concordance were set at 95% sensitivity (positive concordance) and 90% specificity (negative concordance) when compared to the RT-PCR gold standard.We handled missing data using the missing indicator approach.

Ethical consideration
The National Health Research Ethics Committee (NHREC) approved the validation protocol and all procedures before the initiation of this study (NHREC/01/01/2007-14/12/2020).All procedures followed the harmonized protocol for the study and were in line with the WHO Research and Development priorities and blueprint for the novel coronavirus [22].Written informed consent was obtained from all participants.

Description of RDT diagnostic results
Table 6 summarises the diagnostic performance of six antigen-based RDT kits (Lifotronic, Lumira Dx, Mologic, MP SARS, Panbio, and SGTI-Flex) for COVID-19 diagnosis/screening compared to the gold-standard test (RT-PCR) in terms of area under ROC curve (AUROC) values, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).Apart from Lumira Dx with an AUROC value of 0.79 (95% CI: 0.74-0.83), the other antigen-based kits had AUROC values that were less than 0.7.
A combined forest plot of the sensitivities and specificities of RDTs evaluated in this study is presented in the S1 Table .which spanned the last quarter of 2021 and early 2022.Thus, the differences in sensitivity may be related to the difference in the circulating variants at the time of the study compared to the variant from which the reagents were produced.Antigen detection tests are generally less sensitive and less likely to pick up very early infections compared to molecular tests.In the USA, the FDA identified certain EUA-authorized antigen tests whose performance may be impacted by mutations in the SARS-CoV-2 and some of these tests were modified accordingly [24].
Several studies have described reduced sensitivity of Ag-RDTs to the alpha, delta, and omicron variants even after adjusting for other conditions affecting sensitivity such as Ct values and the presence of clinical symptoms [25][26][27][28].It is worthy to note that the predominant circulating variants in this study were Delta prior to December 2021 and Omicron after that.On the other hand, more recent studies have implicated immunity from vaccination and previous infection as opposed to virus variants as a reason for the decreasing sensitivity of Ag-RDT [29].Thus, variation in the seroprevalence levels in the different study countries may account for the marked differences in sensitivity observed.The relative proportion of asymptomatic persons, persons with extended time post-symptom onset, and low viral loads may also have contributed to the combined low sensitivity values obtained in this study; this is reflected in the higher sensitivity observed in symptomatic versus asymptomatic individuals after subgroup analysis.
In general, diagnostic accuracy data from Africa show lower sensitivity than reports from Europe and America [30].Khandker et al. [30] have speculated that the performance of these tests may be affected by freeze-thaw cycles (for reagents requiring cold chain) during transportation to Africa and Asia from foreign manufacturers.They proffered local manufacturing as the solution to the possible deterioration associated with transportation.It is also important to note that the RDT kits detect the antigen and not the nuclear material, which is the target for the PCR test (Gold standard).The nuclear material (viral RNA) may be detected even when a viable virus is not present, unlike the situation with a viral antigen which is an indication of the presence of a viable virus.Also, viral antigens are subject to the immune response of the host.These reasons may explain the differences in the detection rate by RDTs and PCR and thus the lower sensitivity of the RDTs.
Four antibody-based RDTs were also evaluated for their suitability for COVID-19 diagnosis in this study.Their overall poor performance is not surprising as antibody tests measure the body's immune response to the virus rather than the presence of the virus itself which is the target of the gold standard PCR.In the early months of the pandemic, and against a background of absent immunity to a new disease, antibody tests were widely evaluated as diagnostic tools for COVID-19 [31].However, their usefulness was limited by delayed and poor predictability of the timing of antibody appearance in the serum.This limitation has been further compounded by the build-up of immunity from past infections and vaccination as the pandemic progressed.In our study, nevertheless, the specificity of the IgM component of antibody-based RDTs was generally higher than that of IgG, implying that this aspect of the tests may have some utility in detecting recent infections and may find use in orthogonal testing algorithms.A study carried out by WHO collaborating centre in Belgium reported cross-reactivity of SARS-CoV-2 antibodies of 20.3%, 18.5% and 7.5% with malaria, schistosomiasis, and dengue respectively; however, this cross-reactivity was not associated with detecting the SARS-CoV-2 antigen [32].While we do not have specific data for each of the study's locations, malaria (20 70%) [33], schistosomiasis (44.8% to 71.5) [34], and dengue (30.8%) [35] are prevalent in Nigeria.However, our study did not investigate their possible effects on the performance of the various assays.

Limitations
A major strength of this study is its multi-centred nature which makes the data representative of the entire country.The study, however, has a few limitations, which include: (1) Due to the need to minimize contact with suspected SARS-CoV-2 exposed individuals, some demographic data and clinical symptoms were collected using self-administered questionnaires with the resultant 'non-response' (2) a comprehensive head-to-head comparison of the Ag-RDTs was not feasible because some enrolees did not consent to the collection of multiple swabs; (3) comparability of the results may be limited by the non-uniformity in SARS-CoV-2 assays employed for molecular diagnosis at the various sites; and (4) RT-PCR tests, despite being the gold standard, is known to detect a non-viable viral nucleic acid.This may result in an erroneous perception of low performance in the RDTs.

Recommendations
Limitations notwithstanding, findings from our study have implications for the use of rapid diagnostic tests for COVID-19 diagnosis in Nigeria.Antibody-based RDTs may have limited use in detecting acute disease but they may be useful in the surveillance of recent and past COVID-19 infections, this is supported by the higher sensitivity values obtained for symptomatic individuals in this study.Ag-RDTs may be used to quickly detect COVID-19 at reduced cost in clinical settings but negative results must be interpreted cautiously and supplemented with additional PCR testing where clinical suspicion is high.The use of these tests as screening tools at international ports of entry, campgrounds, and mass gatherings is not recommended, especially during periods of high transmission, as many positive persons would be missed resulting in untoward transmission.
In summary, the RDTs evaluated are not sensitive enough to replace RT-PCR for the diagnosis of COVID-19.However, the high specificity of Ag-RDTs makes them useful supplements capable of reducing the need for molecular testing in confirming COVID-19-positive samples in Nigeria.

Further research
More studies are required to evaluate the effect of temperature, other prevailing infections such as malaria, virus strains, and population immunity on the performance of these tests across diverse settings.The need for the local development of SAR-CoV-2 RDTs with improved sensitivity and specificity is also recommended.

Table 3 . List of PCR reagents supplied by NCDC.
using commercially available RT-PCR kits targeting at least three genes (RdRp, N, ORF1b, and E genes) of the virus.The samples were analysed on a real-time PCR machine using the cycling conditions recommended by the respective kit's manufacturer.The human housekeeping gene RNAse P was targeted as the internal control for both nucleic acid extraction and PCR for the normalisation of cycle threshold (Ct) values.The Ct values of < 40 were considered positive while values � 40 were considered negative according to the manufacturer's instructions.Ag and four (4) Ab RDT kits were evaluated in this study; all kits were tested in all study sites.All Ag RDTs were lateral flow immunochromatographic tests (LFT) except the Lumira Dx SARS-CoV-2 Ag test (Lumira Dx UK), which is a fluorescent immunoassay (FIA) with a digital machine for detecting and reading the results.All the six Ab RDTs were LFTs capable of detecting and differentiating IgM and IgG.A list of RDTs evaluated, including the target antigen/antibody is found in Table2.All samples were blindly tested by two different experienced laboratory scientists.The tests were carried out and interpreted according to the manufacturers' instructions.Nasal swab samples collected for the study were tested for the presence of SARS-CoV-2 antigen while plasma or serum samples were tested for the presence of SARS-CoV-2 antibodies (IgG and IgM) using the various antigen and antibodies detection RDTs respectively.

Table 5 . Clinical symptoms of the study participants (N = 1310).
screening compared to the gold-standard test (PCR).Overall, all the IgM components of the kits had poor discriminative capacity for COVID-19 diagnosis, with AUROC values ranging from 0.51-0.59.Notably, SD Biosensor had the highest AUROC of 0.59.The sensitivity of the IgM detection kits was generally low, with Sugentech having the highest value of 30.0%(95%:

Table 6 . Diagnostic accuracy of antigen-based RDT kits for COVID-19 diagnosis/screening compared to PCR test for combined participants.
Area under the receiver operating characteristics curve; PPV = Positive Predictive Value; NPV = Negative Predictive Value ‡ Calculation of predictive scores requires complete observations for both RDT and PCR, hence the variations in sample size https://doi.org/10.1371/journal.pgph.0003371.t006